Modeling Healthcare Data using Markov Decision Process

Outline Objective Background: Stochastic tools used in healthcare MDP in healthcare Preliminaries Optimality Equations and the Principle of Optimality Solving MDPs Examples References Objective: To discuss the construction and evaluation of Markov Decision Process (MDP) To investigate the role of MDP in healthcare. To identify the most appropriate solution techniques for finite and infinite-horizon problems. To compare MDPs to Standard Markov Model simulation (also known as SMMs) by solving the problem of the optimal timing of living-donor liver transplantation. Medical treatment decisions are often sequential and uncertain. This complexity requires the use of more advanced modeling techniques. Initially, the most common methodology used to evaluate decision analysis problems was the standard decision tree. However, standard decision trees have serious limitations in their ability to model complex situations, especially when outcomes or events occur (or may occur)over time (Roberts, 1992). Because of this limitation, standard decision trees are usually replaced with the use of SMMs to model recurrent health states and future events. The use of Markov models has grown substantially in Medical Decision Making (MDM) since the description of these methods by Beck and Pauker (Beck and Pauker, 1983). But, these models cannot be used to represent problems in which there is a large number of embedded decision nodes in the branches of the decision tree (Detsky, 1997), which often occurs in situations that require sequential decision making. Simulation models based on SMMs is also computationally impractical since in healthcare there are possibly large number of possible embedded decision, or decisions that occur repetitively over time (Alagoz, 2010).